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ABSTRACT 

Realtime crowdsourcing research has demonstrated that 
it is possible to recruit paid crowds within seconds by 
managing a small, fast-reacting worker pool. Realtime 
crowds enable crowd-powered systems that respond at 
interactive speeds: for example, cameras, robots and in- 
stant opinion polls. So far, these techniques have mainly 
been proof-of-conccpt prototypes: research has not yet 
attempted to understand how they might work at large 
scale or optimize their cost/performance trade-offs. In 
this paper, we use queueing theory to analyze the re- 
tainer model for realtime crowdsourcing, in particular its 
expected wait time and cost to requesters. We provide an 
algorithm that allows requesters to minimize their cost 
subject to performance requirements. We then propose 
and analyze three techniques to improve performance: 
push notifications, shared retainer pools, and precruit- 
ment, which involves recalling retainer workers before a 
task actually arrives. An experimental validation finds 
that precruited workers begin a task 500 milliseconds af- 
ter it is posted, delivering results below the one-second 
cognitive threshold for an end-user to stay in flow. 

INTRODUCTION 

Crowdsourcing is no longer constrained to slow, offline 
tasks. Just as traditional programming evolved from 
offline batch processes to realtime results and interac- 
tion, crowdsourcing is now transitioning from wait times 
of hours (Ipeirotis 2010) to seconds (Bernstein, Brandt, 
Miller & Karger 2011). Techniques that place workers 
on active retainer can now recruit crowds in two sec- 
onds (Bernstein et al. 2011), complete traditional crowd- 
sourced votes in five seconds, and maintain continuous 
control of remote interfaces (Lasecki, Murray, White, 
Miller & Bigham 2011). These realtime crowds open the 
door to deployable applications that react intelligently, 
including smart cameras, robot navigators, spreadsheets, 
and on-demand graphic design. 

However, existing realtime techniques have largely been 
prototypes aimed at demonstrating feasibility — they did 
not attempt to understand how these approaches would 
work at large scale or to optimize cost /performance 
trade-offs. We focus on the retainer model, which pays 
workers a small extra wage to be on call while they pur- 
sue other tasks, then respond quickly when a realtime 
request arrives (Bernstein et al. 2011). Currently, the 
retainer model is not optimized for cost or performance, 
nor do reqiiesters have any analytic framework to under- 
stand the relationship between retainer pool size, cost, 
and response time. 
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This paper analyzes the retainer model using queueing 
theory (Gross & Harris 1998) to understand its perfor- 
mance at scale, in particular the trade-off between ex- 
pected wait time and cost. We introduce a simple algo- 
rithm for choosing the optimal size of the retainer pool 
to minimize total cost to the requester subject to the re- 
quester's performance requirements: maximum expected 
wait time or maximum probability of missing a request. 
We then propose several improvements to the retainer 
model that reduce expected wait time. First, retainer 
subscriptions allow workers to sign up for push notifica- 
tions for recruitment, which reduces the length of time 
it takes to recruit new workers onto retainer. Second, 
combining retainer pools across requesters leads to both 
cost and wait time improvements. Large retainer pools 
can then be made more effective by using task routing 
to connect appropriate workers to the tasks that need 
them. Third, a precruitment strategy recalls workers 
from retainer a few moments before a task is expected 
to arrive, dramatically lowering response time. We per- 
form an early empirical evaluation demonstrating that 
precruitment results in median response times of just 
500 milliseconds. 

Our analysis carries several benefits. First, realtime 
tasks can now directly minimize their cost for a given 
performance requirement. Second, the retainer subscrip- 
tions allows workers to register for the tasks they like 
best and have them delivered, rather than constantly 
seeking out new work. Third, we demonstrate empiri- 
cally that these techniques can overcome previous limits 
of "crowds in two seconds" to deliver the feedback to 
the Msci within 500 milliseconds — finally under the one- 
second cognitive threshold for an end-user to remain in 
flow (Nielsen 1993). 

We begin by surveying rcilatcid work on realtime crowd- 
sourcing and wait times in crowdsourcing systems. We 
then describe the retainer model and use queueing the- 
ory to analyze and optimize wait time and cost. We 
introduce our improvements to the model — retainer sub- 
scriptions, global retainer pools, and precruitment — and 
integrate them into our analysis. Finally, we discuss 
limitations of our approach and point to future work 
realizing the vision of realtime crowds. 

RELATED WORK 

Crowdsourcing researchers have a strong interest in fast 
task completion times. Paying more will lead to more 
work completed (Mason & Watts 2009), but not at re- 
altime speed. QuikTurKit introduced two techniques 
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to improve response time: repeatedly posting tasks so 
as to stay visible in the recent task list, and keeping 
workers primed through old tasks until a new task is 
ready (Bigham, Jayant, Ji, Little, Miller, Miller, Miller, 
Tatrowicz, White, White & Yeh 2010). The retainer 
model builds on QuikTurKit by paying workers a small 
fee and notifying them when work is ready, recruiting 
crowds in two seconds (Bernstein et al. 2011). Workers 
can also maintain continuous realtime control of an inter- 
face by electing temporary leaders (Lasecki et al. 2011). 
We contribute a more thorough analysis of the tech- 
niques in these systems and an algorithmic approach to 
helping these systems achieve target wait times at min- 
imum cost. 

Accurate models of crowdsourcing platforms help us un- 
derstand the underlying processes and predict behavior 
when parameters change. Queueing theory has been 
used to estimate throughput and wages on Mechanical 
Turk (Ipeirotis 2010). Survival analysis is another pop- 
ular model for predicting task completion time, espe- 
cially in non-realtime scenarios (Faridani, Hartmann & 
Ipeirotis 2011). We model crowdsourcing task arrival 
processes as Poisson; empirical data suggests that the 
Poisson approximation is accurate when parametrized 
by time of day (Faridani et al. 2011). 

RETAINER MODEL 

The retainer model is a recruitment approach for real- 
time crowdsourcing. This model was introduced for real- 
time interfaces like instant feedback votes and a crowd- 
sourced camera shutter (Bernstein et al. 2011). It pays 
workers a small wage to be on call and return quickly 
when a task is ready. These workers accept the task in 
advance and are paid extra to keep their browser window 
open. While they wait for the task to arrive, workers are 
free to work on other tasks. There are many methods 
for recalling the worker; Bernstein et al. (2011) used a 
modal dialog and an audio alert. Evaluations demon- 
strated that workers messaged in this way begin work in 
two to three seconds. 

QUEUEING THEORY ANALYSIS 

In this section, we investigate a mathematical model of 
retainers. This model allows us to predict how long real- 
time tasks will need to wait. To begin, suppose each task 
type has its own set of retainer workers. When a task 
comes in, a worker leaves the retainer pool to work on 
the task and the retainer system recruits another worker 
to refill the pool. The goal is to maintain a large enough 
pool of retainer workers to handle incoming tasks. In 
other words, we want to minimize the probability that 
the retainer pool will be empty (no retainer workers left), 
subject to cost constraints. The risk is that a burst of 
task arrivals may exhaust the retainer pool before we 
can recruit replacement workers. 

We will model this problem using queueing theory. In 
queueing theory, a set of servers are available to handle 
jobs as they arrive. If all servers are busy handling a 



job when a new job arrives, that job enters a queue of 
waiting tasks and is serviced as soon as it reaches the 
front of the queue. In our scenario, tasks are jobs, and 
retainer workers are servers. 

In this paper, we will consider a class of algorithms that 
set an optimal retainer pool size. Suppose the retainer 
pool is c workers. As jobs come in and remove workers 
from the retainer pool, assume that the system always 
puts out enough requests for new workers to bring the 
pool back to c. That is, if there are cq workers in the 
pool, the system has issued c — cq outstanding requests. 
If, when a job arrives, the pool is empty, the system 
sets it aside for special processing: it directly recruits 
a worker, not for the pool, but for that job. In effect, 
a user with a diverted job is immediately alerted that 
the system is over capacity and the job will be handled 
out-of-band after a short delay. This final assumption 
may not accurately reflect how a running system would 
work, but it provides an upper bound on expected wait 
time and makes it easier to analyze the probability that 
a task will be serviced in realtime. 

Suppose that tasks arrive as a Poisson process at rate A, 
and retainer workers arrive after they are requested as 
a Poisson process at rate /ij^ Then, the empty spots in 
the retainer pool, each of which will become filled when 
a worker arrives, can be thought of as busy machines 
occupied with a job whose completion time is a Poisson 
process with rate fi. In our setup, we also divert jobs 
that arrive when all machines are busy. 

In other words, this is an M/M/c/c queue where jobs 
arrive at rate A and have processing time A ba- 
sic M/M/1 queue assumes Poisson arrival and comple- 
tion processes, a single server, and a potentially infinite 
queue. An M/M/c/c queue has c parallel machines in- 
stead of one, and rejects or redirects requests when there 
are no servers to immediately handle the incoming re- 
quest (Gross & Harris 1998). Imagine a telephone sys- 
tem, for example, that gives a busy signal if all c lines 
are busy. The meaning of /i has changed slightly to indi- 
cate worker recruitment time instead of a job completion 
time, but the mathematical analysis is the same. 

To optimize performance, we need to understand the 
probability that all workers are busy, since that is the 
case where a job has to wait (for expected time l//x). 
We also need to understand the cost of having a retainer 
pool of size c. Since the system pays workers propor- 
tional to how long they are on retainer without a job, 
the total cost is proportional to the average number of 
idle machines — these are the ones representing workers 
waiting on retainer. Finally, we will eventually need to 
integrate worker abandonment into our model, since not 
all workers respond to the retainer alert. 



^ These assumptions are perhaps overly ideal. Job arrivals on 
Mechanical Turk are heavy-tailed (Ipeirotis 2010). However, 
much of our analysis is independent of the arrival distribu- 
tion, and systems can always substitute empirically observed 
distributions and solve numerically. 
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Probability of an Empty Pool 

The probability that a job must wait can be derived 
using Erlang's loss formula (Gross & Harris 1998). We 
set p, the traffic intensity, to be the percentage of system 
resources that are are required to service newly incoming 
tasks: p = X/fi. In M/M/c/c queueing systems, as we 
will demonstrate, p < c is necessary for the system to 
keep up with incoming requests. 

The probability of an empty retainer pool (all c "servers" 
busy) is Erlang's loss formula: 



7r(c) = 



(1) 



A remarkable property of Erlang's loss formula is that 
this relationship requires no assumptions about the dis- 
tributions of job arrival time or worker recruitment time, 
in particular whether they are Poisson. It only depends 
on the means /i and A. 



Expected Waiting Time 

For some applications, the probability of a task needing 
to wait is less important than the expected wait time 
for the task. The two quantities are directly related. 
The expected wait time is the probability of an empty 
retainer pool multiplied by the expected wait time when 
the pool is empty, or i7r(c): 



1 1 pVd 



(2) 



This expression gives us a direct relationship between 
the size of the retainer pool, the arriving task and worker 
rates, and the expected wait time. 

As a sanity check: when A — > (few arrivals) we have 
p — > in which case 7r(c) — > 0[^ In other words, we are 
very unlikely to have an empty pool so the expected wait 
time also goes to zero. This relationship is visualized in 
Figure [l];c). 



Expected Cost 

Once we understand expected waiting time, we can ana- 
lyze the retainer model's cost characteristics. Bernstein 
et al.'s (2011) experiments suggested that workers could 
be maintained on retainer for $0.30 per hour at a rate of 
^(t per minute, but this analysis is fairly simplistic. To 
understand cost more completely, we need to know the 
expected number of workers on retainer. 

The probability of having i busy servers in an M/M/c/c 
queue is a more general version of Erlang's loss formula: 



Tr{i) 



(3) 



We can derive the closed form expression of the expected 
number of busy servers: 



m 



= p 
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(4) 



In steady state, we need to pay all retainer workers who 
are not busy. That is, we expect to have c — p{l — 7r(c)) 
workers waiting on retainer. If our retainer salary rate 
is s (e.g., s = ^(t), we would pay s(c — p(l — 7r(c))) per 
unit time on average. 

Visualizing the Relationships 

While these equations give us precise relationships, they 
may not convey intuitions about the performance of the 
platform. Figure [T] plots these relationships for several 
possible values of p. These figures show a knee in the 
curve aX c~ p for getting a good probability of response. 
A pool size c> p means that an empty pool's overall rate 
of recruitment of workers, c^, exceeds the arrival rate of 
tasks. In other words, we begin to catch up and rebuild 
a set of available workersj^ On the other hand, if c < p, 
then even an empty queue will not recruit workers fast 
enough to cover all arriving tasks, so it will stay empty^ 

Figure [2] visualizes the relationship between the re- 
quester's cost and the probability of waiting. We derive 
this parametric curve by choosing values of c, then find- 
ing the cost and probability of waiting given that value. 
Paying more (i.e., for a larger pool) always improves the 
probability that the system can immediately handle a 
request. However, for small values of p, e.g. p < 1, pay- 
ing 1-1.5(1; per minute brings the probability of waiting 
near zero. When tasks arrive quite quickly, 2.5(1; or more 
is necessary to achieve similar performance. 

OPTIMAL RETAINER POOL SIZE 

A queueing theory model allows us to determine the 
number of workers to keep on active retainer. The size 
of the retainer pool is typically the only value that re- 
questers can manipulate, and it impacts both cost and 
expected wait time. Requesters want to minimize their 
costs by keeping the retainer pool as small as possible 
while also maintaining a low probability that the task 
cannot be served in realtime. In this section, we present 
techniques for choosing the size of the retainer pool. 

Our goal is to find an optimal value of c, given 1) the ar- 
rival rates A and p, and 2) desired performance, in terms 
of the probability of a miss 7r(c) or total cost. We assume 
that the requester knows A and p either through empir- 
ical observation or estimation. We also assume that A 



^Actually, 7r(c) jc 



■^When p/c — >■ 0, the number of free workers goes to c — p(l — 
p"^ /c\), or effectively c. 

*As p oo, the number of free workers goes to c — pc/{p + 
c) — c(l — p/(p -|- c)) which goes to 0. 
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(a) Cost of retainer 



(b) Probability of waiting 



(c) Expected wait time 
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Figure 1. Graphs that visualize the relationships between retainer pool size, traffic intensity, and (a) cost, (b) probability 
of a task waiting, and (c) expected wait time. In the graph of expected wait time, we set A = 1, so ^ = . When p > c, 
there are often not enough workers on retainer to service all tasks. As a result, wait time goes up, but cost goes down. 
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Figure 2. By calculating cost and the probability of a 
task needing to wait for integer values of c G [1, 15], we can 
visualize the relationship between the two values. 

and fi are constant, but it is enough just for them not to 
change too quickly. 

One approach to finding c is to specify the maximum al- 
lowable expected wait time for a task, or (equivalently) 
the maximum allowable probability that an incoming 
task will not be served in realtime. The intuition for 
this approach can be seen in Figure [ijb): if p = .5, for 
example, and the requester wants a less than 5% proba- 
bility of any given task needing to wait, then c = 3 is the 
smallest retainer pool that can make such a guarantee. 

Algorithmically, if Pmax is the maximum desired proba- 
bility of a task not being served in realtime, we want to 
minimize c subject to 7r(c) < Pmax- To find the solution, 
we use a binary search over possible values of c. 

A more interesting version of the problem is for the re- 
quester to attach a dollar value to each task that cannot 
be serviced in realtime. For example, some pizza deliv- 
ery companies do not charge the customer for the pizza 
if they cannot deliver it within thirty minutes. A miss 
then costs the company the value of the pizza plus the 
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Figure 3. By assigning a dollar value to missed tasks, we 
can visualize the relationship between retainer size and 
total cost. Assuming traffic intensity p = 1 and retainer 
wage s = 1, these curves demonstrate the trade-off be- 
tween more missed tasks on the left part of the graph 
and higher retainer costs on the right. 

deliveryman's wage spent delivering the late pizza. A 
requester might similarly offer the service for free if it 
is not completed in realtime, or they might decide that 
the bad experience of a non-realtime result is worth $1 
in lost potential revenue from that user. 

It now becomes possible to directly minimize the re- 
quester's total cost. Let Ctotai be the expected total 
cost to the requester and Ctask be the loss if a task is 
not completed in realtime. Then Ctotai is the sum of 
the expected task cost — zero if addressed in realtime, or 
Ctask otherwise — and the wage for the retainer workers 
derived from Equation [4j 



Ctotai = CtaskT^{c) + s(c - p{\ - 7r(c))) 



(5) 



We can minimize this total value. Figure |3] shows this 
curve for several possible values of Ctask when p = 1. 
The minimum value on the y axis for each curve is the 
optimal retainer size. 
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WORKER ABANDONMENT 

Queueing theory models assume that a server wiU al- 
ways begin a job once it is assigned. However, workers 
will sometimes leave the computer, close the window, 
or otherwise not respond to the retainer alert. Empir- 
ically, Bernstein et al. (2011) found that about 10-20% 
of workers on active retainer never responded. 

Our model can be adapted to capture worker abandon- 
ment. Let a be the percentage of workers who abandon 
the task, that is, they do not return after the retainer 
alert. A straightforward edit is to add the constant a to 
the probability that a task will not be serviced in real- 
time, so that probability is now a + 7r(c). The response 
to this would be to increase c to cover the difference and 
recall 1/a workers for each job instead of 1. However, 
this is a conservative approach. 

A more cost-effective approach would be to alert another 
worker if the first worker does not respond quickly. If 
the mean worker response time to an alert is R, choose 
a scalar a and wait until aR for the worker to respond. 
If the worker has not responded by then, the platform 
immediately alerts another worker and waits another 
aR seconds before issuing a third request. There is a 
constant probability of a worker responding within time 
aR, so the expected number of alerts before getting a 
response will likewise be a constant. 

Unfortunately, queueing theory cannot easily accommo- 
date this kind of approach. A model including server 
breakdown is a close match, except that server repair 
recruits another worker, which means that task arrivals 
are correlated and no longer Poisson. To bound the ex- 
pected cost within the queueing theory framework, we 
envision a more complicated construction, which would 
be unlikely to be used in a running system. A sketch 
of the proof follows. We maintain several tiered retainer 
pools. If a worker in tier i does not respond within time 
aR, we alert a worker in tier i + 1. Task arrivals to each 
queue are now Poisson, since a constant fraction of the 
requests to tier i will pass through to i-|-l. For example, 
we might choose a such that half of the requests will 
respond in time. Then, if tier i has task arrival rate A, 
tier i + 1 would have arrival rate A/2. We would see a 
sequence of geometrically decreasing pool sizes, meaning 
the total cost, a geometric sum, will be a small multiple 
of the first-tier cost, which we have already analyzed. 

IMPROVEMENTS TO THE RETAINER MODEL 

So far, we have analyzed the original form of the retainer 
model. However, by extending it, we can improve its 
performance considerably. In this section, we introduce 
three changes to the retainer model and analyze their 
impact: retainer subscriptions, globally shared retainer 
pools with task routing, and predictive recruitment. 

Retainer Subscriptions 

The worker arrival rate n is a, limiting factor of the 
retainer model: previous experiments on MTurk saw 



/i « |, or 1 worker every 6 seconds. A small arrival 
rate means that the retainer pool can take a long time 
to fill, which is particularly problematic for large bursts 
or tasks that need multiple simultaneous workers. 

One way to increase /i is for the platform to put together 
a panel of retainer subscribers who can be directly no- 
tified when the retainer pool needs to recruit a replace- 
ment. The insight behind this approach is to change 
from a pull model of crowdsourcing, where workers seek 
out tasks, to a push model, where tasks offer themselves 
to workers. Workers could subscribe to a task type, so 
that when the platform needs a retainer worker for a task 
of that type, the platform could send a dialog notifica- 
tion to one or more subscribed workers and offer them 
the opportunity to complete one task in the next few 
minutes. Workers who accept are now on retainer, can 
continue working on other tasks, and will be interrupted 
whenever the realtime task arrives. 

A push notification is likely to reduce the time it takes 
to recruit a worker onto retainer, thereby increasing fi. 

Global Retainer Pools 

In the previous analysis, each requester maintained their 
own retainer pool. In this section, we analyze how shar- 
ing one global retainer pool across requesters improves 
performance. We also investigate how to route tasks to 
workers in a globally pooled retainer. 

Global Pool Analysis 

Another way of writing Equation [T] the probability of 
a missed task, is tt{c) — 7r(0) • p'^/d, where 7r(0) = 
CLUoP'/^-y^ (Gross & Harris 1998). Recall Stirling's 
approximation that c! ~ V^nc^c/e)''. Also note that 
the sum that defines 7r(0) is decreasing geometrically, 
so we can approximate 7r(0) ~ a constant. This 

approximation gives us: 

7r(c) w e-P^^iep/cY (6) 

If we have k different tasks each with traffic intensity 
p and queue size c, the probability of an empty pool is 
roughly kn{c) sa ke~P\/2TTc{ep/cY: we multiply by k 
because each requester independently suffers. 

Now suppose we bring all the retainer pools together, 
creating one "superpool" of size kc. The task arrival 
rate A increases by k but the rate at which we recruit one 
worker p remains unchanged. Thus the traffic intensity 
increases by a factor of k to kp. So, the probability of 
an empty pool with combined retainers is 

e-''PV2^{ep/c)'"' = V2^ {e-P{ep/cyf (7) 

Ignoring the square root factor, we see the main term 
being exponentiated by a factor of k. In other words, 
the loss rate declines exponentially with the number of 
retainer pools we bundle. 

We can look at some approximations for these results. 
Suppose we set c = (1 + e)p, just above our c ~ p knee 
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in the curves from Figure [1] Then, with a single retainer 
pool, 7r(0) is about 




This is the same quantity as shows up in the typical 
analysis of the upper tail of the Chernoff bound. There, 
we generally approximate this quantity as e"*^ ^1'^ ^ which 
is reasonably accurate for any e < 1. In short, the prob- 
ability of an empty pool is roughly ''1'^ . 

Using this approximation, we can ask what retainer pool 
size in the globally shared case will yield the same empty- 
pool bound e~'^'Pl'^ as we found in the singular case. 
As we argued above, moving to the globally shared case 
multiplies p by a factor of fc. Since the exponent we 
care about proportional to we can decrease e by 

a factor of \fk and end up with the same bound as the 
singular case. In other words, the fraction e of "buffer" 
workers that we need in our retainer pool is proportional 
to \/fc, as compared to the factor fc in the singular case. 
We thus need many fewer extra workers per extra task: 
much like standard error decreases by a square root fac- 
tor as sample size increases, we have less uncertainty in 
arrival rates as more requesters join together. 

Task Routing 

Shared retainer pools introduce speed and cost improve- 
ments, but workers will subscribe to multiple realtime 
task types and can only work on one realtime task at a 
time. This situation immediately raises the question of 
how to decide which worker should be assigned to each 
retainer pool when a spot opens up. Market forces like 
task pricing will help solve this problem, but microtask 
markets like Mechanical Turk are very clustered on a 
small number of prices (often 2 — 5(1;). Inefficient task 
routing could lead to logjams where certain tasks can- 
not find workers. In this section, we demonstrate that 
a straightforward approach like uniform randomization 
could lead to extremely slow response times, and we in- 
troduce a linear programming solution that optimizes 
response times across tasks. 

Suppose we have a set of task types T = ti, . . . , i„, and 
tasks of type tj arrive with Poisson distribution and rate 
Xj. Not every worker can complete every task: workers 
may have only signed up to be on retainer for particular 
task types, or they may not have the qualifications for 
all task types. We split workers into groups wi , . . . , Wm 
that are uniquely identified by the tasks that group can 
complete. So, for example, w\ might represent all the 
workers who are on retainer for ti, ti, and ^3. We say 
that W is the set of all worker types {W = wi, . . . , Wm), 
and that each Wi has a Poisson arrival rate fj,i. 



Worker arrival 
rate ^. 







t ^ ) 









Task arrival {^~^\ 

. \^_^ \ y 

ti \ S 

Figure 4. A task routing scenario where a typical ran- 
domized approach would lead to poor results. t\ would 
receive relatively few workers. Depending on the values 
of /ii, each task type could find itself in this starved state. 

Given a set of task types T, a set of worker types W, 
and arrival rates for each, our goal is to assign workers 
to tasks to maximize the throughput of the system. To 
do so in steady state, we need to decide how many worker 
arrivals — more precisely, what portion of the overall ar- 
rival rate — from each group should be assigned to each 
task. Let us say that the rate at which workers from 
group Wi should be assigned to tasks of type is a^-. 
These assignments must sum to the total arrival rate of 
the worker group: X^JLi ^ij — l^i- For example, in our 
earlier example of Wi, if = 1, one possible assignment 
is Oil = .5, ai2 = .25, 013 = .25. 

A standard approach would be to assign each worker ar- 
rival randomly to one of the task types that he or she can 
complete. (That is, aij are equal for any i.) However, 
this approach could result in slow completion times. In 
Figure |4j has four times the arrival rate of Wi or W2- 
Random assignment would send workers to at rate 
1/4-1-1 = 5/4, whereas ti would receive workers at just 
1/2. Depending on which workers are online, each of the 
task types could find itself in a similarly starved state. 

Instead, a centralized system can route workers to min- 
imize expected wait time. This goal can be described as 
a linear programming problem, but in fact can be solved 
using maximum flow, which is significantly faster than 
general linear programming. The following constraints 
suffice to define a linear programming problem — they 
indicate that the incoming worker rate to each task type 
is at least as high as the incoming task rate, and that all 
the worker assignments from a worker group sum to no 
more than the arrival rate of that worker group. 

aij > Xj for all j, 

' (9) 
/ ciij < fJ-i for all i- 
j 

We can also choose to be more specific about the quan- 
tity to maximize. For example, as we have seen above, 
task wait times are typically a function of the ratio of 
arrival rate and service rate (A//i), known as traffic in- 
tensity p. We can define an analogous p here to be 
the ratio of the incoming task arrivals to the summed 
rate of arrivals from all worker groups for that task: 
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p — '^j/J^i^ij- We then minimize its worst case across 
all tasks: 

minimize p 

subject to py Gij > Xj for all j, 

i (10) 

ttij < p.i for all i. 

j 

By merging retainer pools, the platform can thus help 
guarantee fast results for all tasks. 

Scaling 

One practical difficulty with this approach is estimating 
p,i as the number of task types grows. If there are |r| 
different task types, there are 21-^' different combinations 
of task types that a worker can sign up for, and thus 
\W\ = 2l^l. This set is an extremely large number of ar- 
rival rates to try and estimate accurately, and will make 
the linear program hard to solve because there will be 
an exponential number of constraints. 

However, the problem of efficient feature representation 
is a common one in machine learning. There are many 
approaches to this problem. We may find that in practice 
only a small number of task type combinations can occur. 
We can also enforce this, for example by setting a ceiling 
on the number of task types a worker can subscribe to 
at once. With a limit of two subscriptions, \W\ = |Tp 
instead of 2l^l. 

Precruitment: Predictive Recruitment 

Previous research was limited by the length of time it 
took a worker to respond to the retainer alert. However, 
our model suggests that even this limit of "crowds in two 
seconds" (Bernstein et al. 2011) is unnecessary, and that 
crowds could be recruited effectively instantaneously. 

The insight behind our solution is precruitment: noti- 
fying retainer workers before the task actually arrives. 
The queueing theory model involves estimating 1/A, the 
expected length of time before the next task will arrive. 
If 1 /A is about the length of time it takes to recall a re- 
tainer worker, we can recall a retainer worker and expect 
to have a task by the time the worker arrives. As we will 
demonstrate, workers are also happy to wait at a "Load- 
ing..." screen even if the task is not ready immediately. 

Workers take 2-3 seconds to arrive (Bernstein et al. 2011) 
and will wait for roughly ten seconds afterwards (Nielsen 
1993). The Poisson task arrival process has rate A, and 
Poisson distributions have standard deviation y/X. So, 
the platform can precruit A -I- /3\/A workers per second 
for upcoming requests, where /3 is a slack variable that 
controls how many extra standard deviations to precruit 
for safety. Any workers who do not have tasks within 
a predetermined wait time would need to be paid and 
dismissed. However, as the platform becomes large, the 
standard deviation will become proportionally smaller 
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Figure 5. The median length of time between the mole 
image appearing and the workers moving to click on it was 
0.50 seconds. So, a platform can recall retainer workers 
early and get crowds in half a second instead of waiting 
for the workers to respond to the retainer alert. 

relative to the mean, making it possible to waste very 
little money on extra workers. 

In fact, the entire precruitment system can be repre- 
sented as its own M/M/c/c queueing system. Many of 
the same techniques introduced earlier can be applied to 
help optimize the size of a precruitment pool in relation 
to the standard retainer pool. 

Evaluation 

We ran a study on Mechanical Turk as a proof-of-concept 
for precruitment. In the study, we followed the protocol 
of Bernstein et al. (2011) by offering three cents for a one- 
minute retainer task: a game of Whack-a-Mole. After 
waiting on retainer for one minute, workers responded 
to the retainer alert and were asked to quickly click on 
the picture of a mole randomly placed in a 3x3 grid of 
dirt mounds. However, after responding to the alert and 
before the mole appeared, workers needed to wait for 
a randomly selected length of time between and 20 
seconds while a "Loading..." indicator displayed. 

We measured the length of time between the appear- 
ance of the mole and: a) mouse movement in the di- 
rection of the mole, and b) the click on the mole. We 
discarded any responses where worker clicked on a dirt 
mound instead of the mole or where the browser did not 
record millisecond-precision timing. After filtering, our 
dataset consisted of fifty workers who completed N=373 
Whack-a-Mole tasks. One limitation of our design is that 
Whack-a-Mole is a relatively enjoyable task, and workers 
might not be so attentive for less game-like tasks. 

The median length of time between the mole's appear- 
ance and the worker moving the mouse toward the mole 
to click on it was 0.50 seconds across all wait times (mean 
0.86, std. dev. 1.45, Figure [5|. The median length of 
time before clicking on the mole was 1.12 seconds (mean 
1.87, std. dev. 2.23). There is a negligible correlation be- 
tween wait time and mouse movement delay [B? — .001), 
suggesting that workers react roughly as quickly right 
after they arrive as they do twenty seconds later. 

We can use the same dataset to compare precruitment 
to the retainer approach presented in Bernstein et al. 
(2011). This comparison is possible because a "Load- 
ing..." delay of zero seconds is the exact same worker 
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experience as the standard retainer model. We are in- 
terested in the lag time betwcien the task arriving and 
mouse movement to whack the mole. Here, a new task 
results in an alert being sent to the worker, so we start 
our timer with the alert. Without precruitment, the me- 
dian time between task posting and mouse move was 
1.36 seconds (mean 1.41, std. dev. 0.30). 

This result suggests that, had wc used a standard re- 
tainer model with this task, we would have seen mouse 
movement typically after 1.36 seconds. Using precruit- 
ment, we get mouse movement in 0.5 seconds. Precruit- 
ment finally breaks through the sub-second cognitive 
barrier that keeps users in flow (Nielsen 1993). 

DISCUSSION 

Our model has several limitations. One limitation is that 
an M/M/c model may be a better match for certain re- 
tainer implementations where the idea is to handle tasks 
FIFO and not immediately give up on realtime response 
for tasks when the pool is empty. Second, worker recall 
delays depend on the length of time the worker has been 
waiting on retainer (Bernstein et al. 2011), but our anal- 
ysis ignores this fact. Third, our model assumes that it 
can always recruit new retainer workers into the pool, 
but the retainer population is limited in practice. How- 
ever, we believe that these observations can be integrated 
into our optimizations. 

One empirical question we have not addressed is the 
number of workers that need to be on a crowdsourc- 
ing platform to make sure that requesters can maintain 
full retainer pools. This number also depends on the 
percentage of workers who are willing to sign up for re- 
tainer tasks. Since the retainer model pays more than 
batch tasks, we anticipate that this percentage will be 
high. On Mechanical Turk, our experience is that it is 
not difficult for a single requester to get twenty or thirty 
workers on retainer simultaneously. However, as more 
requesters use retainers, these dynamics may shift. 

While realtime retainers are the motivating example in 
this paper, the entire Mechanical Turk platform can be 
thought of as a large retainer system where workers are 
paid zero retainer wage and the worker recall rate is ex- 
tremely slow, since workers return on their own initiative 
rather than by recall. Precruitment is another kind of 
retainer model queue where workers are recalled before 
the task even arrives. All three queues could be analyzed 
together as a queueing network in order to more effec- 
tively understand the entire system. However, it is also 
possible to bound the probability of a slow task response 
via the probability that any of the retainer pools are 
empty. Supported by our results so far, we suggest that 
queueing theory can be applied for many other problems 
in the space of realtime crowdsourcing as well. 

Our analysis suggests that paid crowdsourcing platforms 
could integrate a globally-managed retainer into their 
design. This will not only change the types of crowd- 
sourcing that are common, but will also introduce new 



elements of worker reputation. We suggest two new 
reputation statistics. First, a worker's median response 
time characterizes how quickly they respond to the alert 
and begin working on a retainer task. Requesters pre- 
fer workers with low response times. Second, workers 
are tagged with a response rate: the percentage of the 
time that they successfully respond to a retainer alert. 
If a worker does not respond to the alert within a given 
length of time (e.g., five seconds), the system finds an- 
other person and the worker is not paid. 

CONCLUSION 

In this paper, we have analyzed and optimized the re- 
tainer model for realtime crowdsourcing. We applied 
queueing theory to demonstrate specific relationships be- 
tween task and worker arrival rates, the size of the re- 
tainer pool (workers waiting for a task), cost and ex- 
pected wait time. We introduced a technique for choos- 
ing the optimal retainer pool size given a requester's 
needs, and for integrating abandonment into the queue- 
ing theory model. Finally, we described three new tech- 
niques that improve the performance of the retainer 
model: retainer subscriptions, shared retainer pools, and 
precruitment, or recalling retainer workers before the 
task arrives. These techniques suggest directions for 
future platform development, and have already shown 
promise in returning results to users in 500 milliseconds. 
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